Abstract Colorectal cancer is the 3rd most common newly diagnosed cancer and the 3rd most common cause of cancer death among US men and women. Neoadjuvant chemoradiation therapy (CRT) followed by total mesorectal excision is the standard of care for locally advanced rectal cancer. Preoperative CRT has clearly improved rates of local disease control and colostomy free survival; however the response to therapy is heterogeneous. It would be very useful to be able to predict the individual risk of each patient, so that their therapy can be personalized. The goal of our study is to derive clinically useful radiomic signatures from multimodal imaging data for the early prediction of treatment outcomes in rectal cancer patients. Central to our methodology are 1) an improved deep learning model for automatically segmenting tumors from multimodal imaging data with high accuracy; and 2) a multi-task deep learning model for robustly learning informative radiomic features to predict survival and recurrence. These methods will be used to derive individualized predictive indices of treatment outcomes based on a multimodal imaging dataset of rectal cancer patients who have received preoperative CRT. Our methods are generally applicable to radiomic studies of cancer patients. All methods will be made publicly available and form an important new resource for the broader radiomics community.